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An examination of the literature of the last fifteen years reveals the degree to which 
"communication competence" has become a significant referent with respect to the goal of 
communication instruction. This is especially evident in si e n publications and resources of 
the Speech Communication Association as Developing Communication Competence in 
Children (Allen and Brown, 1976), Development of Functional Communication 
Competencies: Pre-K to Grades 6 and Grades 7-12 (Wood, 1977), Assessing Functional 
Communication (Larson, et al., 1978), Communication Competency Assessment Instrument 
(R. B. Rubin, 1982), Speaking and Listening Competencies for High School Graduates 
(1982), Communication for Careers: Oral Communication Competencies Needed by 
Community College Graduates Entering Careers (1982), Communication Competencies for 
Teachers (1988), and the more recent Wingspread Conference Proceedings: Essential College 
Sophomore Speaking and Listening Competencies (Quianthy, 1990). Further examination of 
these and other sources also reveals the degree to which communication competence and 
assessment are intrinsically intertwined. 

The most recent and strongest statement by the Speech Communication Association 
and communication educators concerning the importance of communication competence 
and the need for appropriate assessment strategies was provided both by the convening and 
the results of the 1990 Summer Conference on the Assessment of Oral Communication 
Skills. One of the many charges pursued by the conference participants was the 
development of a "public speaking skills portion of a test of oral communication" 
(Backlund, 1990, p. 5). This was the charge given to one group of conference participants. 
This group, consisting of coromunication scholai-s from eleven universities throughout the 
United States, was further charged to take a position "on which skills should be assessed, 
what criteria should be used to assess them, and suggested procedures" (Backlund, 1990, p. 
5) and to consider developing and recommending appropriate assessment prototypes. To 
date, the efforts of this group have resulted in the development of The Competent Speaker, 
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a public speaking assessment instrument designed for the college sophomore (grade 
fourteen) level. 

The purpose of this paper is to explain the need for a public speaking assessment 
instrument, the process employed by the group in developing The Competent Speaker 
instrument, and the nature and characteristics of the instrument. Other papers and 
materials provided in this short course will provide preliminary data concerning the 
reliability and validity of the instrument, describe the variety of ways in which the 
instrument might be used for both pedagogical and assessment purposes, and demonstrate 
application of the instrument as an assessment tool. 

The Need for a Standardized Public Speaking Assessment Instrument 

The need for a standardized public spealdng assessment instrument has become 
increasingly evident as research concerning communication competence generally, and the 
assessment of communication competence specifically, have progressed in recent years. 
Despite the many issues of controversy surrounding the nature of communication 
competence, there are three central issues which have emerged from the research and 
about which there is increasing agreement-issues particularly germane to the theme of this 
paper. 

The first issue concerns the nature of the components of communication 
competence. The literature reveals a great degree of controversy over the years concerning 
this question. The primary controversy concerns which of three broad components, 
corresponding to Bloom's (1964a, 1964b) taxonomy of cognitive, affective, and 
psychomotor domains, comprise communication competence. Of these, there appears to 
be consistent agreement that competence consists of at least knowledge or cognitions and 
behavior (Backlund & Wiemann, 1978; Cooley & Roach, 1984; Fogel, 1979; Harris, 1979; 
McCroskey, 1982, 1984; Spitzberg, 1983; and Wiemann and Backlund, 1980). In other 



words, communication competence requires both a repertoire of skills and a body of 
knowledge concerning those skills and their implementation. 

The affective domain has increasingly been included as a third component. The 
position here is that communication competence consists not only of knowing what and 
how to perform, but also valuing performance sufficiently to do so (Moore, 1981). This 
component, more frequently referred to as motivation, was identified by R. B. Rubin 
(1983) as the third dimension of competence, by Spitzberg and Hecht (1984) as integral to 
their model of relational competence, and by Spitzberg and Cupach (1989) as a component 
of interpersonal competence. Moreover, to the degree that such predispositions as 
communication apprehension (McCroskey, 1970, 1977), receiver apprehension (Wheeless, 
1975), and willingness to communicate (McCroskey and Richmond, 1987) are related to 
communication competence (Clark, 1989), the role of motivation, or the affective domain, 
is well supported. 

While there appears to be general consensus that communication competence 
consists of knowledge, skills, and motivation, there is much less clarity concerning the 
specific components which comprise each of these domains. Typically, researchers have 
attempted to identify the specific communicafion behaviors or skills which comprise 
competence. This has resulted in an extensive and highly diverse list of behaviors, from 
broad behaviors such as empathy, behavioral flexibility, interaction management, listening, 
and speaking (Bochner and Kelly, 1974; DiSalvo, 1980; Reardon, 1987; and Wiemann, 
1977) to more specific behaviors such as articulation and pronunciation (Duran, 1983; R. B. 
Rubin, 1982). As Spitzberg (1987) points out however, "Given a lack of conceptual 
guidance, measurement efforts to date have yet to identify a comprehensive or consistent 
set of competence components" (p. 5). This relates directly to the second issue central to 
the nature of communication competence. 

Is communication competence a trait or a state? At one level, the question here is 
whether communication competence is a cross-situational disposition or whether it is 



dependent upon the situation. While some studies suggest a few traits, such as rhetorical 
sensitivity and communication apprehension, exhibit cross-situational consistency, most 
research indicates that competence appears to be too situationally bound and its research is 
too fraught with methodological problems to posit communication competence as a general 
disposition (Andersen, 1987; R. B. Rubin, 1990: Spitzberg, 1987). This is further supported 
by the quantity of research concerned with identifying competent communication behaviors 
in such diverse settings as interpersonal (Bochner and Kelly, 1974; Parks, 1985; Spitzberg 
and Hecht, 1984), group (Bradley, 1980), public (Quianthy, 1990), organization (DiSalvo, 
1980; Monge, Bachman, Dillard, & Eisenberg, 1982; Papa, 1989), mass media (Anderson, 
1983) and intercultural (Chen, 1988; Cooley & Roach, 1984). As R. B. Rubin (1990) states, 
"Research continues to point to a need to use both state and trait measures to examine 
communication competence until v/e have a firm understanding of which measures assess 
traits and which estimate state-influenced behaviors" (p. 104). This is consistent with 
Spitzberg and Cupach's (1989) view that "both trait and state approaches to the 
conceptualization of competence seem viable and even compatible" (p. 53). 

The third issue central to this discussion, is frequently characterized as the 
"effectiveness vs. appropriateness" debate regarding competency. While there appears to 
have been general consensus that appropriateness is directly related to competence, the 
relationship between competence and effectiveness has not been as consistently clear. 
Sjme conceptualizations are not vt ry explicit about the relationship between effectiveness 
and competence. Others, however, state quite explicitly that effectiveness is a fundamental 
criterion of competence. 

Representative of those who are not explicit about the relationship between 
effectiveness and competence are Allen and Brown (1976), who view communication 
competence as "an awareness of the transactions that occur between people" (p. 248). 
While this perspective ties competence "to actual performance of language in social 
contexts" (p. 248), nowhere in the explication of the four principal features of competence 
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is it clearly tied to effectiveness. Similarly, Harris (1979) is less specific about the role of 
effectiveness in her definition of interpersonal competence as "the ability to create and 
coordinate interpersonal systems" (p. 32). McCroskey (1982), however, is quite explicit in 
his statement that communication effectiveness "is neither a necessary nor sufficient 
condition for a judgment of competence" (p. 3). 

Bochner and Kelly (1974) and Heath (1977) suggest only a general link between 
effectiveness and competence, but others are much more specific. Weimaim (1977), for 
example, defines communication competence as the ability of an interactant to accomplish 
goals "while maintaining the face and line of his fellow interactants within the constraints of 
the situation" (p. 198). Fogel (1979) expresses an even stronger link between effectiveness 
and communication competence by defining the latter as the ability "to affect another's 
attitudes, behaviors, and beliefs' (p. 15). Spitzberg (1983), on the other hand, argues that 
both appropriateness and effectiveness are essential to competent communication. 

Despite the different emphases of various definitions, both effectiveness and 
appropriateness appear to have a role in communication competence. As R. B. Rubin 
(1990) explains, "Early distinctions between competence (appropriateness) and 
effectiveness (goal accomplishment) seem to have faded. Most researchers today agree 
that both elements must be present" (p. 109). 

To summarize, there appears to be widespread agreement that communication 
competence consists of at least three dimensions (cognitions/knowledge, affect/motivation, 
behaviors/skills), is both a general disposition and context dependent, and requires 
behavior which is both effective and appropriate. Each of these issues has implications for 
the assessment of communication competence. 

Perhaps the clearest and most important implication is the virtual impossibility of 
developing a single instrument which can provide a valid and reliable assessment of 
communication competence. The first barrier is provided by its multi-dimensional nature. 
While the cognitive and affective dimensions potentially could be assessed by a single 
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paper-and-pencil instrument, the behavioral dimension could not-requiring a two-part 
instrument/process at a minimum. However, the second barrier, the contextual nature of 
communication competence, is insurmountable. Again, while it may be possible to develop 
a single instrument to assess the cognitive and affective diuiensions of the primary 
communication contexts (dyadic, group, public, mediated, etc.), a single instrument or 
procedure to assess the behavioral dimension of all contexts would not be possible. 

This conclusion is consistent with that of the participants of the SCA Summer 
Conference on Speech Communication Eiducation who, as reported by Backlund (1990), 
stated in their final resolution that "No single communication assessment instrument can 
meet all of the objectives of effective assessment, and it would be unproductive to spend 
time and energy on developing such an instrument" (p. 1). As Backlund further explains, 
"While many individuals claim a need for a national test of speaking and listening skills that 
would reliably and validly assess these skills, and a wide variety of people have indicated 
that such a test would be highly desirable, the conference participants decided that such a 
test would be impractical. The wide variety of factors impinging on the assessment 
procedures, including coverage of the wide range of objectives, cultural issues, issues 
surrounding the levels associated with any test, etc., mitigated against the development of a 
single test." (pp. 4-5). 

Given the impracticality of developing a single instrument to assess communication 
competence, the focus must be on developing multiple instruments or procedures for 
assessing competence within specific contexts. One of the most salient contexts for speech 
communication educators and classrooms, at all levels, is the public speaking context. 

While there are numerous instruments available for evaluating public speaking 
performance (perhaps as many as there are speech communication educators), there are 
none which have a clearly established basis in theory, have been systematically tested for 
validity and reliability, and are designed for application within the standard classroom 
setting. The only instrument which approaches meeting these criteria is R. B. Rubin's 
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Communication Competency Assessment Instrument (1982). However, this instrument is 
designed to provide a more comprehensive assessment of communication competence, of 
which public speaking is one part, and is designed for use outside the typical classroom 
setting. 

Develo pnien * of The Competent Spea/rer Instrument 

After considerable deliberation, the SCA assigned task force decided that the 
greatest need f . »r an instrument which could be used for the following purposes: (!) as a 
pedagogical tool for the evaluation and development of public speaking skills in the 
classroom; and (2) as a pp; (testing out) and/or post (exit) assessment of public speaking 
skills in the basic and/or public speaking course. This further suggested that the 
instrument should be developed for the college sophomore (grade 14) level. This would 
provide an instrument which could be used as a post (exit) assessment at the high school 
(grade 12) level, as well as either a pre (testing out) or a post assessment at the college 
sophomore level. 

The task force also decided that the instrument should be based upon, and an 
extension of, the research concerning both communication competence and public 
speaking assessment. After an extensive review of the literature, referenced earlier in this 
paper and in the task force's bibliography (Moore, 1991), the two documents which 
appeared most germane to the content of a public speaking assessment instrument were 
SCA's Speaking and Listening Competencies for High School Graduates (1982) and SCA's 
Wingspread Conference Proceedings, Communication is Life: Essential College Sophomore 
Speaking and Listening Competencies (Quianthy, 1990). The task force selected the public 
speaking competencies identified in the "college sophomore" document (Table 1) and 
reviewed them against the "high school" document (Table 2) to insure both 
conceptualizations of speaking competency were included. Next, the combined list of 
competencies was compared with the seven public speaking competencies contained in 
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R. B. Rubin's Communication Competency Assessment Instrument (1982). Satisfied that 
the combined list of competencies represented the best and most current thinking within 
the discipline concerning the characteristics c public speaking competence, the task force 
agreed to modification in the rewording of the competencies to insure clarity of language 
and consistency of structure. This process resulted in the eight competency statements 
which comprise The Competent 5pea^er instrument (Tabie 3). 

Table 1 

Speaking Competencies listed in Communication is Life: 
Essential College Sophomore Speaking and Listening 
Competencies^ (Quianthy, 1990). 

- Determine the purpose of oral discourse. 

- Choose a topic and restrict it according to the purpose and audience. 

- Fulfill the purpose of oral discourse by formulating a thesis statement, providing 

adequate support material, selecting a suitable organization pattern, 
demonstrating careful choice of words, providing effective transitions, and 
demonstrating suitable interpersonal skills. 

- Employ vocal variety in rate, pitch, and intensity. 

- Articulate clearly. 

- Employ the level of American English appropriate to the designated audience. 

- Demonstrate nonverbal behavior that supports the verbal message. 



Table 2 

Speaking Competencies 'isted in SCA Guidelines: 
Speaking and Listening Competencies for 
High School Graduates, (1982). 



Use words, pronunciation, and grammar appropriate for situation. 

Use nonverbal signs appropriate for situation. 

Use voice effectively. 

Express ideas clearly and concisely. 

Express and defend with evidence your point of view. 

Organize (order) messages so that others can understand ili-ur,. 

Summarize messages. 
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Table 3 

The Competent Speaker. Eight Public Speaking Competencies 

(Morreale et al., 1990) 



- Chooses and narrows a topic appropriate to the audience and occasion. 

- Communicates the thesis/specinc purpose in a manner appropriate for audience and 

occasion. 

• Provides supporting material appropriate to the audience and occasion. 

- Uses an organizational pattern appropriate to the topic, audience, occasion, and 

purpose. 

- Uses language appropriate to the audience and occasion. 

<• Uses vocal variety in rate, pitch, and intensity, to heighten and maintain interest 
appropriate to the audience and occasion. 

- Uses pronunciation, grammar, and articulation appropriate to the audience and 

occasion. 

- Uses physical behaviors that support the verbal message. 



Following identification of the eight public speaking competencies, the task force 
developed performance standards (criteria) by which each competency could be evaluated 
or measured (see The Competent Speaker. Eight Public Speaking Competencies and 
Standards/Criteria for Assessment, located in Short Course packet). As with the 
competencies, the standards/criteria were based upon the competency literature identified 
earlier, as well as published guidelines for speech evaluation (see, for example, Bock and 
Bock, 1981; Mead and Rubin, 1985; and Powers, 1984). The reader should note that the 
public speaking competencies, and more particularly, the standards/criteria for their 
assessment, are still in a pilot stage of development and testing. 

As explained by Morreale (1990) in an earlier report, following development of the 
eight competencies and corresponding evaluative criteria, the task force generated a 
speech performance evaluation form. The Competent Speaker Speech Performance 
Evaluation Form (located in Short Course packet). In designing the evaluation form, 
earlier published SCA guidelines for constructing a speech evaluation instrument (Boch & 
Boch, 1981, 21-22), were considered, with particular emphasis on scale construction and 
controlling for rater errors. Additionally, National College Board recommendations for 
the development of measures of speaking and listening (Powers, 1984, 7), were observed. 



Moreover, every effort was made to insure the instrument conformed to the "Policy on 
Criteria for the Assessment of Oral Communication," a resolution proposed by a second 
task force of the 1990 SCA Summer Conference on the Assessment of Oral 
Communication Skills and submitted to SCA for approval by appropriate boards and 
legislative committees. See Appendix A for a list of these criteria and a notation of those 
with which The Competent 5peaJi:er instrument complies. Two of the more important 
criteria, those concerning instrument reliability and validity, are currently being addressed 
by the task force, with the preliminary results to be announced elsewhere in this short 
course. 

Significant Characteristics of The Competent Speaker?\ib\\c Speaking Assessment 
Instrument 

As noted previously, every effort has been made to insure that the instrument is 
consistent with both current research concerning the nature of communication competence 
and current policy concerning its assessment. A few of the more significant characteristics 
of The Competent 5/7eaJter instrument relevant to this objective and worthy of special note 
include the following: 

1. Assesses public speaking behavior . In recognition of the inappropriateness, if not 
impossibility, of assessing all domains of competence, the instrument focuses 
exclusively on the assessment of public speaking "behaviors." Tlius, it is not designed, 
nor does it purport, to measure either "knowledge/cognitions" about public speaking 
or "affect/motivation" for public speaking. Regardless, as with the assessment of any 
behavior, the instrument may permit one to make limited inferences about the 
knowledge and motivation of the speaker-inferences limited at most to the speaker's 
knowledge of, and motivations about, public speaking within the target context . 
However, such inferences should be made with caution, since any reliability or 
validity associated with the instrument's assessment of "behavior," does not apply to 
its assessment of "knowledge" or "motivation." 

13 
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2. Assesses both verbal and nonverbal behavior . The instrument identifies competencies 

for both verbal and nonverbal behavior and provides standards/criteria for assessing 
competence in the use of each. However, assessment is limited to the public speaking 
context and should not be generalized to other contexts, e.g., dyadic, group, etc. 

3. Assesses remote preparation skills . The instmment should provide a holistic assessment 

of the speaker's remote preparation skills. However, the assessment of specific 
remote preparation skills or the diagnosis of strong versus weak preparation skills will 
require additional procedures. 

4. Does not assess listening skills . The instrument is designed for the assessment of 

"speaking" bidlls only. Listening skills should be assessed by instruments and 
procedures designed for that purpose. 

5. Provides for either an atomistic or holistic assessment . The eight discreet competency 

statements, with their corresponding standards/criteria, provide for an 
atomistic/analytic assessment. However, neither the competency statements, nor 
their corresponding standards/criteria, prevent assessment based upon a holistic 
impression. 

6. Assesses degree of competence . The standards/criteria for each competency describe 

degrees of competence at the "unsatisfactory," "satisfactory," and "excellent" levels. 
An assessment can be made of either or both the degree to which the speaker 
possesses each competency and the degree to which the speaker possesses public 
speaking competence in general. 

7. Descriptions of "competent" responses provided as anchors . Each of the eight 

competency statements is accompanied by a descriptive statement of the 
standards/criteria for judging each of the three degrees of competence. Where 
appropriate, examples of responses illustrative of each degree of competence are 
provided as anchors. 
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8. Instrument is free of cultural bias . Each competency is assessed with respect to the 
target audience and occasion. In other words, judgments are based upon the degree 
to which the behavior is appropriate to the "audience and occasion." As long as the 
evaluator/assessor bases his/her judgments on these criteria, cultural bias should not 
become a factor. This requires, however, that the evaluator/assessor have a clear 
understanding of the target audience and occasion and uses the standards consistent 
with the target audience and occasion as the only basis for evaluation/assessment. 

Conclusion 

The focus of this Short Course, The Competent Spea/rer Public Speaking Assessment 
Instrument, represents the energy and effort, over the past one and one-half years, of 
eleven different communication educators from institutions of higher education throughout 
the United States and is based upon the cumulated research concerning communication 
competence and oral communication assessment. Despite this, the instrument is still in the 
developmental stage. Therefore, the instrument is not recommended for use as an exit 
instrument or for exemption purposes at this time. Upon completion of appropriate 
research with respect to its validity and reliability, efforts will be made to provide for 
distribution of the instrument through the national offices of the Speech Communication 
Association. In the meantime, as a pedagogical tool, for use within the public speaking 
classroom, it should prove quite helpful. The task force encourages you to use the 
instrument, and the accompanying materials, in your classroom. We especially welcome 
any feedback you can provide concerning the instrument. 
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APPENDIX A 



The left column contains abbreviated descriptions of the criteria contained in the proposed SCA 
"Policy on Criteria for the Assessment of Oral Communication." The right column identifies 



"Policy 

which criteria are met by The Competent Speaker as&tssmeni instrument. 



General Criteria 

1. Assesses knowledge, skills, & 
attitude 

2. Judged by trained assessor 



3. Assesses speaking & listening 

4. Sensitive to assessee's disabilities 



5, Assessment based on atomistic data 
& holistic impression 



Notation of Compliance by 
The Competent 



ipnance t 
Speaker 



Public speaking skills 
only 

Yes (via manual and 
trainmg tape) 

Speaking only 

Yes (via manual and 
training tape) 

Designed for atomistic 
data; holistic possible 



Criteria for Content of A ssessment 

1, Assesses verbal & nonverbal in 
multiple settings 

2. Assesses specialty area of speech 
majors 



Verbal & nonverbal in 
public speaking setting 

Not applicable 



Criteria for Assessment Instruments 

1. Skills assessed by performance in 
public speaking setting 

2. Assesses degree of competence 

3. Identifies range of responses which 
constitutes degrees of competence 

4. Meets acceptable level of reliability 

5. Meets acceptable standards of validity 

6. Free of cultural, sexual, ethnic, 
racial, age & developmental bias 

7. Suitable for developmental level 
of assessee 

8. Standardized procedures for 
administering 



Yes 

Yes 
Yes 

Data being collected 
Data being collected 
Yes 



Yes (college sophomore/ 
Grade 14 level) 

Yes 
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Critena for Assessmerr. Procedures & Administration 

1. Prccedures protect nghts of assessee Yes (procedures outlined 

in manual) 

2. Assessment for procedural decision Use for procedural 
based on multiple sorrccs of info. decisions in manual 

3. Assessors trained by speech Yes (via manual and 
communication professionals training tape) 
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INTRODUCTION 

The Competent Speaker Speech Evaluation Form has been carefully 
developed and psychometr i cal 1 y tested in order to fullv conceotual i ze 
communication competency in public speaking. That development and 
testing have demonstrated the essent i al ness of the instrument's 
appropriate administration. The instructor and/or communication 
professional should attend to the following considerations with regard 
to the instrument: M ) logistics for administration: (2) Dotential 
uses; and (3) cautions for use. 

LOGISTICS FOR ADMINISTRATION 
Training in the Use of the Form 

Before using The Competent Speaker Evaluation Form to rate a public 
speaking performance, the evaluatorfs) first should careful Iv studv the 
instrument, the eight competencies it contains, and most imoortant the 
standards/criteria for each competency. The evaluator also should 
become familiar with the descriptions of unsat i sf actorv . satisfactory, 
and excellent levels of performance for each competency. After becoming 
familiar with the competencies and criteria, the evaluator should use 
the training videotape of exemplary speeches thuslv: 

First, view the three "anchor speeches." at the beginning of the 
tape, identified as unsatisfactory, satisfactory, and excellent. Studv 
the competencies and criteria in relation to those three videotaoed 
anchors . 

Next, view the twelve "unidentified speeches" (or as many as time 
allows) located after the three anchors. Rate these soeeches on the 
speech evaluation form, without knowing the level of comoetencv that 
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each represents. After rating each soeech. comoare the ratina assianed 
by the evaiuator with the "true ratinq" fnormative ratinq") for that 
speech (the true ratinqs are listed on the outside of the traininq 
tape). When the evaiuator 's ratinqs become similar to the true ratinqs 
for the speeches, he or she mav be considered trained and readv to 
evaluate speeches usinq The Competent Speaker Evaluation Form. 
Using _the Form . to Evaluate and fjrade Speeches 

In the actual use of the form to evaluate speeches, the evaiuator 
may use any numerical weighting svstem for the three levels of 
competency that suits the particular context or course requirements. 
For example, the evaiuator could: (1) simplv examine the level of 
performance for each competency without assigning anv numerical value 
to the performance of the competencv; OR (2) assiqn one point for 
unsatisfactory, two points for sat i sf actorv . or three points for 
excellent, for each competencv (in this case, the ranqe for the grade 
fot the speech would be from 8 to 24); OR (3) for additional speeches 
increase the values assigned to reflect the increasing level of 
importance or value of the particular speech -- ie. for each competencv. 
assign two points for unsat i sf actorv , four points for satisfactory, and 
six points for excellent (the range for the grade would be from 16 to 
48) . 

In addition to using various numerical weighting svstems for the 
three levels of competency, the evaiuator also may consider 
differentially weighting the different competencies, depending on the 
context. For example, certain competenc i es may be deemed more important 
than others in the following situations: 
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1. In a documentat ive or research-based speech. "'ComDstencv Three: 
Provides appropriate supportinq material ," might be assigned more points 
than other competencies. 

2. In a speech immediately following a lecture on style and delivery. 
"Competency Eight: Uses physical behaviors that support the verbal 
message," might be assigned more points. 

3. Following a lecture on the use of languaqe. Competencies Five. Six. 
and Seven might be weighted more heavily than other competencies. 

4. In a persuasive speech, as opposed to an informative speech, the 
evaluator may choose to weight certain competencies higher than others. 

When using the form to evaluate speeches, the eva 1 uator C s mav 
consider the merit and value of videotaping the speech performance. 
Videotaped recording and playback for students has demonstrated success 
in improving language usage and delivery (Miles, 1981: Mulac. 1974"). 
The primary advantage of videotaping is that it allows "us to see 
ourself as others see us" (Dance anc' Zak-Dance. 1986. p. 244'). 

POTENTIAL USES OF THE FORM 

The Competent Speaker Speech Evaluation Form has an array of uses 
and applications including but not limited to the following: 

1. Entrance assessment at the beginning of a course or at the beginning 
of a student's academic tenure at an institution. 

2. Exit assessment at the end of a course or at the conclusion of a 
student's academic tenure at an institution. 

3. Placement assessment of a student among or into classes or courses. 
0 4. Diagnostic assessment, within a class, for prescriptive purposes. 

for providing feedback, and for encouraging development as a speaker. 
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5. Speech evaluation and criticism within a class- 

CAUTIONS REGARDING THE USE OF THE FORM 
Use of The Competent Soeaker Evaluation Form to rate public 
speaking performances necessitates the observance of certain cautions 
(SCA Criteria for Assessment Procedures and Administration (Crocker- 
Lakness • 1990) ; 

1 The evaluator should protect the rights of the speaker /assessee bv: 
(a) oresenting the instrument* explaining how it will be administered, 
and defining its criteria; and fb) maintaining appropriate 
confidentiality of the results and their usefs). 

2. The speaker*s performance end the evaluation of that performance 
should not be used as the "sole** instrument for *'procedural decisions" 
such as placement, exemption, academic credit or grade. Rather, it 
should be used only for evaluating competence in presenting a public 
speech* In order to be used for procedural decisions concerning an 
individual, it should be combined with other sources of information such 
as: fa) direct evidence of actual communication performance in school 
and/or other contexts; fb) results of formal competence assessment: 
and/or fc) measures of coT.mun i cat ion apprehension or avoidance. 

3. The evaluatorfs) using the form should be sat i sf actor i 1 v trained in 
its use, as described under *'Training in the Use of the Form.*' 

4. The evaluatorfs) should be educated about and sensitive to the 
effects of relevant physical, psychological, and cultural biases 
(including gender, ethnic, racial age, and developmental), 

5. The evaluatorfs) should note that numerical weighting, both between 
and within competencies, relates to the psychometric value of the 
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evaluation process. Optimally, the Comoetent Sosaker Evaluation Form 
is designed to be used as a ratio scale. It involves the assignment of 
numbers for the purpose of identifvinq ordered relations of the 
competencies -- the order being arbitrarily assigned -- with eaual 
intervals and an absolute zero point (Williams, 1979). If thi^ evaluator 
uses a checking system and not a numerical weighting svstem for the 
competencies, the psychometric value of the instrument is diminished. 
6. The evaluator(s) should avoid and be educated about rating scale 
usage problems (Bock & Bock. 1981: Rubin. 1991) such as: a) lack of 
interest, in which case the eval uator f s ) mav rate the soeakers 
inconsistently. If evaluators do not conform careful Iv to the 
standards/criteria established for each of the comoetenc i es . thev will 
evaluate students' oerformances unfairly. Vigilance and diligence are 
prerequisite to fair speech evaluation. Evaluators who cannot or choose 
not to evaluate speeches, based carefully on the standards/ criteria of 
The Competent Speaker Form, should be retrained or replaced; fb) 
personal bias, in which case the eva 1 uator f s ) mav be either too easy 
(positive leniency error) or too hard (negative leniency error) on all 
speakers. Or, the evaluator may be too easy (positive halo error) or 
too hard (negative halo error) on a specific student; (c) trait error, 
in which case the evaluatoris) may be either too easv or too hard on a 
given trait (competency) on the evaluation form. This error mav occur 
in the evaluation of one or all speakers if the evaluator(s) attends to 
or neglects certain competencies; and (d) central tendency error, in 
which case the evaluator tends to group scores toward the m-ddle 
(satisfactory) range of the evaluation form. Evaluators tend to avoid 
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making extreme evaluations, either unsatisfactory or excellent. 

To avoid rating scale usage problems, adequate traininq and 
retraining in the use of The Comoetent Soeaker Evaluation Form is 
encouraged. If the form is used in multiole sections of the same 
course, training and testing for inter-rater reliability the deqree 
of agreement among different evaluators or raters when iudaina or 
evaluating the same soeech -- is advised to orovide consistency of 
grading between sections. Statistical tests of inter-rater reliabilitv 
should be administered periodically. 
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Introduction 



The CAT Subcommittee on Public Speaking Competency developed criteria to assess a 
speaker's competence. An eight item instrument was created which empirically defines the 
variable. (See 'The Competent Speaker Speech Performance Evaluation Form".) A nine point 
rating scale is used to judge the level of ability for each competency. Twelve speech teachers 
viewed tapes of twelve speakers and evaluated all speeches on all eight competencies. Thus a 
total of 144 evaluation forms are the data for this report. The FACETS computer program, 
written by John M. Linacre, provides the means of performing a Rasch analysis. 

The model used for this data analysis follows: 

Pnmjgik 

log ( ) = Bn - Cj - Dgi - Fgk 

Pnmjgik - 1 

Bn n = 1 - 12 (speakers) 

Cj j = 1 - 12 O'udges) 

Dgi g = 1, i = 1 - 8 (items) 

Fgk g = 1, k = 1 - 9 (9 point rating scale) 



The Rasch model provides the needed method for analysis. A frame of reference is con- 
structed, and a variable defined. Observations provide the counts necessary to operationalize 
the variable upon a line of inquiry. In 1953 Georg Rasch, a Danish mathematician, developed 
the essentials of this model for item analysis. It has the property that one set of parameters can 
be eliminated by means of conditional probabilities. This allows a researcher to focus on one 
set at a time. Person ability or attitude can be independent of the difficulty of the items on the 
test. The difficulty estimate of the items can also be independent of the people tested. This is 
crucial because it allows sample-free measures. Scores are transformed into common units of 
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measure, "logits". A person's probable response is separated from the items on the instrument. 
This makes it possible to compare groups, or individuals, regardless of the test taken or the 
sample used to calibrate it. (For a more complete explanation, see Tatum, 1991; Wright and 
Masters, 1982; Wright and Stone, 1979.) 

This Rasch analysis performs the following functions: 

• 1) shows the structure of the rating scale 

• 2) determines whether the evaluation form operationalizes the variable "public 
speaking competency" 

• 3) provides a calibration of evaluation items 

• 4) measures the severity of the judges 

• 5) discovers rater inconsistency 

• 6) produces objective measures of speakers' competency. 



Data Analysis 



SCALE STRUCTURE 

A nine point rating scale is divided into three classes; unsatisfactory, satisfactory, and excel- 
lent. The criteria are defined for each class, and the raters assign a number to each spea ir's 
competencies. The scale structure line shows ihat the rating scale itself is working beautifully 
- the categories are in pronounced locations and define nine different levels. (One could 
conceive of them as definite steps on a la'^der.) Often raters have a tendency to group scores 
around the middle of the scale values. McCroskey, Arnold and Pritchard (1967) established 
that the end points on a semantic differential were further from the points next to them than 
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the other points were from each other. Some raters do not like to make extreme judgments. 
The Rasch model calibrates the nonlinearity of the rating scale, thus eliminating this concern. 



Count.?. 1 1 1 I 1 1 

J 8 8 9 4 6 6 2 5 

5 0 5 5 B 0 8 8 3 

+ + ++ + + ++ + 

Score: 0* H 1-— H— 2-H--3-H-4--H-5-H-6— -7 -H -*8 

Label! 1 2 3 4 5 6 7 8 9 

I I I t t I i 

I { I I ( I I 

Logit: -3,0 -2.0 -1.0 0.0 1.0 2.0 3.0 



ITEM ANALYSIS 

Ail items are not created equal. That is, a range of difficulty must be covered if a test is to be 
useful in measuring any variable. The "ruler" which measures the variable must be calibrated 
in equal units, or "logits". The items which comprise this ruler lie upon a line of inquiry. They 
are centered on zero. Items that arc easy to do are negative numbers, those t''at are harder 
are positive. 

Map of Items 

The following is a visual represeniation of the items defining the variable. The items cover a 
reasonable range. This demonstraies that the evaluation form does a good job of defining 
public speaking competency. 



Item Map 



Easiest Thesis 

Appropriate topic 
Appropriate language 

Articulation 

Appropriate organizational pattern 

i 
I 

Vocal variety 
; Supporting material 

Hardest Nonverbal behaviors 



Item Statistics 

The following table provides statistical information for each item. The items are listed in 
measure order from most difficult to easiest. Count is the raw score for the item. (The computer 
program rescores the rating scale from 1-2-3-4 to 0-1-2-3.) Sample is how many people 
responded to the item. Calibration is the item's measure, or its placement upon the line of 
inquiry, expressed in logits. Error is how accurately the item is measured. Mean square 
(MNSQ) represents the expected responses lo the item. If an item has a high mean square, 
then there is a question whether it fits upon the line of inquiry, and is of help in defining the 
variable. A mean square of 1.0 is exactly what is expected; .7 to 1.4 is normal. For example, a 
mean square of 1.6 for an item means there is 60% more noise than expected, or 2.4 is 140% 
more than expected. The standardized fit statistic is the sureness of the item's fit. A rule of 
thumb is to look closely at anything over a mean square of 1.5 and a fit of 3 to determine the 
usefulness of the item. The items on the evaluation form fit the pattern of expected responses. 
This means all the items are on the line of inquiry and contribute to the definition of the 
' variable. 
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Item Measurement Report 











Calib. Hodel 


Infit 


Outfit 




N Iteas 


Scor9 


Count 


Averaae 


Locit Error 


HnSa Std 


HnSa Std 


! N 


1 appropriate topic 


639 


144 


4.4 


-0.14 0.06 


1.1 1 


l.l 0 


! 1 


2 thesis 


642 


144 


4.5 


-0.15 0.06 


LI 0 


1.1 0 


I 2 


3 supporting aaterial 


555 


144 


3.9 


0.13 0.06 


1.1 0 


l.l 0 


! 3 


4 appropriate org pattern 


589 


144 


4.1 


0.02 0.06 


1.0 0 


1.0 0 


1 4 


5 appropriate language 


636 


144 


4.4 


-0.13 0.06 


0.6 -4 


O.b -4 


1 5 


6 vocal variety 


556 


144 


3.9 


0.12 0.06 


0.9 0 


0.9 0 


6 


7 articulation 


622 


144 


4.3 


-0.09 0,06 


0.8 -2 


0.9 -1 


7 


8 nonverbal behaviors 


516 


144 


3.6 


0.25 0.06 


1.2 2 


1.2 1 


8 



f 



RATER ANALYSIS 

Facets computer program allows a researcher to separately analyze various components of the 
situation under investigation. Raters are an important element of public speaking evaluation. 
It is necessary to examine their behavior for a complete understanding of the assessment of 
public speaking competency. 

Raw scores must be conditioned before they can be used to define and measure a variable 
along a line of inquiry. When scores are conditioned, and transformed to measures, they no 
longer are bound by their idiosyncrasies. Raw scores are not linear because there is no equal 
distance between them. They only have meaning in conjunction with the Rasch model. 
Together the score and the model determine the probabilities. The "bad" properties can be 
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fixed by the use of the Rasch model. Raw scores do not have the property of additivity, which 
is a requirement for measurement. Measures must have scalability - equal measures calibrated 
upon the line of inquiry. 

At. example of disparity in the meaning of raw scores can be shown in judges' evaluations of 
a person's public speaking competence. Judges 1,2,3, are tough raters. Judges 4.5,6, are much 
easier raters. If the scores from the first three judges are "4", "5", "6", and the other group also 
scores "4", "5", "6", the average from each group of judges is the same - "5". However, a score 
of 5 from tough judges is "worth" more than a 5 from easy judges. Without the linear 
transformation of the Rasch model, the scores cannot reflect this difference. 

Research conducted by members of the MESA Psychometric Laboratory at the University of 
Chicago over the past ten years reveals that judges,, no matter how well trained, do not rate 
alike. In fact, it is not even desirable to attempt to force judges into one common mode. As we 
know from communication theory, every person has his or her own perceptual world, and 
attends to different details. Bock and Bock (1981) discuss four general types of speech raters. 
In a Rasch analysis we assume each rater's individuality and are not concerned with inter-rater 
reliability as an end to itself. It is only one of many indicators. Rather, it is the consistency with 
which the judge uses the evaluation form that is important. A Rasch analysis will adjust for the 
different types and severity of raters as long as they share a common understanding of the 
evaluation form and are individually consistent in their use of the rating scale. 



Rater Measurement Report 

The following is a visual representation of rater severity as well as a report of rater statistics. 
The standard error is .07 and the adjusted standard deviation is .30. 
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Rater Measurement Report 



Rater Measure MNSQ Fit 

Tough 



11 


0.68 


0.9 


0 


13 


0.22 


0.8 


<• 

-1 


10 


0.15 


1.9 


5 


01 


0.09 


1.1 


0 


03 


0.09 


0.8 


-1 


04 


0.09 


1.0 


0 


05 


0.06 


0.6 


-3 


09 


0.00 


1.3 


2 


02 


-0.21 


0.9 


0 


12 


-0.25 


0.8 


-1 


08 


-0.42 


1.2 


1 


06 


-0.51 


0.4 


-5 



This table demonstrates the wide variation in severity of the twelve raters. The Rasch model 
adjusts for the toughness of the judges, and calibrates them in common units of measure. This 
makes it possible to compare individual as well as groups of judges. The mean square and fit 
statistics reveal only one inconsistent judge - number 10. The mean square of 1.9 denotes 90% 
more noise than expected, and the fit of 5 means we are really, really sure of this. Remember, 
rater 10 is not inconsistent with the others, but with himself. Raters 5 and 6 have mean squares 
and fits that are almost too quiet. They are close to Guttman-like in their consistency. Their 
evaluations hold no surprises or randomness. 
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SPEAKER ANALYSIS 



Now that it has been determined there is a recognizable, measurable variable we can look at 
speakers and their abilities and characteristics. 



Speaker Measurement Report 



More Able 


Speaker 
11 


Measure 
1.23 


MNSQ 
1.4 


Fit 
2 


i 


04 


0.65 


1.2 


1 


• 




n fit) 


1 7 


1 
1 




03 


0.50 


1.3 


2 




08 


0.23 


0.8 


-1 




10 


0.10 


0.6 


-3 




02 
05 


-0.11 
-0.13 


0.7 
0.8 


-2 
-2 




01 


-0.17 


0.9 


0 




09 


-0.20 


1.2 


1 


1 

! 

1 


06 


-0.77 


1.0 


0 


Less Able 


07 


-0.96 


0.8 


0 



The above is a visual representation of the speakers' competence. The mean measure is 0.08, 
the standard error is .07, and the adjusted standard deviation is .58. 

Since they spread out along the line of inquiry, it is possible to examine the differences in 
speakers' competence. The evaluation form is a reliable test of separating speakers into strata 
of ability, and the form provides quite accurate measurement. Because competence is reported 
in common units of measurement, comparisons of more or less ability among speakers is 
feasible. We can say that speaker 11, with a measure of 1.23 logits is twice as competent as 
speaker 12, who has a measure of 0.60. Speakers 6 and 7 are considerably below the group's 
average ability; approximately three quarters and a whole logit respectively. All speakers 
performed as expected. That is the mean squares and fit statistics do not indicate any speaker 
who has an unexpected pattern of competency. 



Conclusions 

The Competent Speaker Speech Performance Evaluation Form is a viable instrument for 
assessing public speaking ability. The items cover a range of the variable and are all along the 
line of inquiry. The raters used the form effectively, and the speakers demonstrated various 
levels of ability. 

Now that the pilot test of the evaluation form is completed, more data must be gathered to 
confirm its usefulness general use. Rasch analyses will enable us to determine if there is 
gender or ethnic bias inherent to the instrument, and if it is also applicable at the high school 
and community college level. The preliminary results are encouraging, and the expectation is 
The Competent Speaker Speech Performance Evaluation Form will be approved by The 
Speech Communication Association for national distribution and use. 
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ABSTRACT 



"THE COMPETENT SPEAKER": Development, reliability, and validity 
of a communication-competency based speech evaluation form. 

At the SCA 1990 Summer Conference on Communication 
Competency Assessment, a subcommittee of 11 speech communication 
professionals was assigned a charge "to develop the public 
speaking skills portion of a test of oral communication." The 
subcommittee reviewed literature regarding speech evaluation and 
oral communication competency and developed a pilot speech 
performance evaluation form, THE COHPETENT SPEAKER. The 
criterion-referenced evaluation form contains eight public 
speaking competencies and attendant criteria for performance of 
those competencies, at unsatisfactory, satisfactory, and 
excellent levels. 

This paper reports results of preliminary psychometric 
testing of THE COMPETENT SPEAKER FORM and plans for future 
testing. In a test using 12 speech communication professional as 
raters, high inter-rater reliability coefficients, ranging from 
.90 to .94, are reported. Other reliability studies are 
underway, using graduate teaching assistants as raters and 
"faulty" or non-speech communication instructors as raters. In 
regard to validity of the speech evaluation form, content 
validity is argued based on the instrument's conceptual base in 
the communication competency literature. Content validity also 
will be tested using a Q-sort of the instruments' criteria and 
competencies. Convergent validity is being tested by correlating 
students' scores using the speech evaluation form with (1) scores 
on the pubic speaking items of the Communication Competency 
Assessment Instrument, m and with (2) the public speaking portion 
of the Personal Report and Communication Apprehension. Finally, 
other studies are planned to evaluate the instrument in terms of 
cultural bias. 
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INTRODUCTION 

Impetus for the genesis of "The Competent Speaker" 
evaluation form originated during the SCA 1990 Summer Conference 
on Communication Competency Assessment. At that conference, a 
subcommittee of the Committee on Assessment and Testing was 
challenged to: 

...work on the public speaking skills portion of a test 
of oral communication. A position should be taken on 
which skills should be assessed, what criteria should 
be used to assess them, and suggested procedures. 
Perhaps prototypes should be developed and recommended 
(Backlund, 1990, p. 1). 

Eleven geographically-dispersed communication scholars determined 

this charge to be the development of a speech performance 

evaluation tool, grounded in and driven by the communication 

competency paradigm, as defined by past and current communication 

literature (see Morreale, 1990, for further explication). 

Specifically, the present instrument focuses on public speaMng 

skills as opposed to knowledge/cognition or motivation/affect. 

The instrument ultimately was developed because of the lack of a 

standardized and tested speech evaluation tool available for 

national distribution. 

Although debate about the very nature of communication 

competency abov^ds, there appears to be some agreement that it 

does include at least three components in line with Bloom's 

(1964a, 1964b) classifications of the cognitive, affective, and 

psychomotor domains (Allen and Brown, k 1976; Spitzberg, 1983). 

And while no one list of communication competencies has received 



ERIC 



43 



4 

general concordance, work in this area progresses as exemplified 
by the speaking and listening competencies developed earlier and 
expanded during the SCA-sponsored 1987 Wingspread Conference 
(Quianthy, 1990) . However, these competencies did not include 
attendant criteria/ standards by which to evaluate the articulated 
performance. 

Before the 1990 Summer Conference on Assessment, 
subcommittee me?nbers worked independently, reviewing existing 
published public speaking instruments and literature on the 
evaluation of pubic speaking. Subsequent to the Conference, the 
task force identified eight public speaking competencies and 
articulated criteria/standards by which to evaluate each 
competency (see Appendix A: Competencies and Criteria) . These 
eight public speaking competencies are consistent with and 
derived from those defined in Communication is Life; Essential 
college sophomore speaking and listening competencies (Quianthy, 
1990) (see Table 1) , and SCA Guidelines; Speaking and li stening 
competencies for high school graduates (1982) (see Table 2). The 
eight competencies, ultimately included in "THE COMPETENT 
SPEAKER" instrument, (Table 3) were derived and refined from the 
competencies in Tables 1 and 2. 
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Table 1 



Speaking Competencies listed in "Communication is Life; Essential 
college sophomore speaking and listening compet encies" (Quianthy, 
1990) 

o Determine the purpose of oral discourse. 

o Choose a topic and restrict it according to the purpose and 
audience. 

o Fulfill the purpose of oral discourse by formulating a 
thesis statement, providing adequate support material, 
selecting a suitable organization pattern, demonstrating 
careful choice of words, providing effective transitions, 
and demonstrating suitable interpersonal skills. 

o Employ vocal variety in rate, pitch, and intensity. 

o Articulate clearly. 

o Employ the level of American English appropriate to the 
designated audience. 

o Demonstrate nonverbal behavior that supports the verbal 
message. 



Table 2 

Speaking Competencies listed in "SCA Guidelines; Spea king and 
listening competencies for high school graduates" (1982). 

o Use words, pronunciation, and grammar appropriate for 
situation. 

o Use nonverbal signs appropriate for situation. 

o Use voice effectively. 

o Express ideas clearly and concisely. 

o Express and defend with evidence your point of view. 

o Organize (order) massage so that others can understand them. 

o Summarize messages. 
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Table 3 

"The Coif.?3etent Speaker; Eight Public Speaking Competencies 
(Morreale. 1990) . 

Competency One 

CHOOSES AND NARROWS A TOPIC APPROPRIATELY FOR THE AUDIENCE 
AND OCCASION 

Competency Two 

COMMUNICATES THE THESIS/ SPECIFIC PURPOSE IN A MANNER 
APPROPRIATE FOR AUDIENCE AND OCCASION 

Competency Three 

PROVIDES APPROPRIATE SUPPORTING MATERIAL BASED ON THE 
AUDIENCE AND OCCASION 

Competency Four 

USES AN ORGANIZATIONAL PATTERN APPROPRIATE TO THE TOPIC, 
AUDIENCE, OCCASION, & PURPOSE 

Competency Five 

USES LANGUAGE THAT IS APPROPRIATE TO THE AUDIENCE AND 
OCCASION 

Competency Six 

USES VOCAL VARIETY IN RATE, PITCH, & INTENSITY, TO HEIGHTEN 
& MAINTAIN INTEREST 

Competency Seve n 

USES PRONUNCIATION, GRAMMAR, AND i\RTICULATION APPROPRIATE TO 

THE DESIGNED AUDIENCE 

Competency Eight 

USES PHYSICAL BEHAVIORS THAT SUPPORT THE VERBAL MESSAGE 



A further review of the literature provided information for 
development of the criteria/ standards by which to evaluate each 
of the eight competencies (Bock & Bock, 1981; Mead & Rubin, 1985; 
Powers, 1984) . Finally, in accordance with the charge, from the 
SCA 1990 Conference on Communication Competency Assessment, a 
speech performance evaluation form needed to be generated. The 
conversion of the eight identified competencies resulted in the 
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development of the "Competent Speaker" Speech Evaluation Form 
(Appendix B) . A pilot version of The Competent Speaker 
evaluation Form and criteria has undergone initial testing for 
reliability and validity. 

PSYCHOMETRIC METHODOLOGY 

Subjects 

In order to test for reliability and validity of "The 
Competent Speaker" evaluation form and criteria, a videotape with 
student speeches in an actual classroom environment at a Mid- 
Western University was developed. The student speechs were 
informative presentations lasting approximately 5 minutes in 
duration. The selection of student speeches began with a group 
of six Graduate Teaching Assistants (GTAs) selecting 
approximately 40 videotaped classroom speeches from the previous 
semesters' presentations. Utilizing the form and criteria the 
speeches were rates by these GTAs as wither unsatisfactory, 
satisfactory, or excellent. From this initial pool of 40, a 
selection of 12 student speeches was made by the public speaking 
course director and one of the GTAs, four at each level of 
competency. These 12 speeches were transferred to one master 
tape for training and rating purposes. The student sample 
represented on the rating video was mixed by gender, five females 
and seven males, and by ethnicity, nine Whites, one Black, one 
Hispanic, and one Philippine. Utilizing a table of random 
numbers these speeches were randomly orlered 12 different times 
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and placed on tapes that were sent to 12 speech communication 
professionals at 12 different universities. Additionally, one 
speech exemplary of each level of competency was selected to 
serve as an anchor by which raters could become familiar with 
performance at each level of competency, before rating the sample 
of 12 speeches. The three anchor speeches were placed at the 
beginning of the master tape and were identified as to the level 
of competency each represented. 
Raters 

The 12 raters were speech communication professional 
teachers at colleges and universities in the U.S. Nine of the 
raters held a Ph.D. or equivalent while three of the raters held 
Masters degrees. The raters' experience in teaching ranged from 
four years to 25 years as estimated by the date of receipt of the 
raters' terminal degrees. Raters were eight females and four 
males, 11 of which were angle and one hispanic. 

Raters received a packet containing instruction for self- 
training on the use of the speech evaluation form and criteria 
and the tape with the 12 student presentations. Specifically, 
the rates were instructed to : (1) review the standards and 
criteria for the competencies before viewing any speeches; (2) 
view the three exemplary speechs while simultaneously reviewing 
the standards and criteria; and (3) view each of the 12 speeches 
without making any formal evaluation, review the standards and 
criteria as they pertained to that speech, and finally, view the 
speech one more time and enter the evaluation on the form. 
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Reliability 

Overall inter-rater reliability for the students' total 
score on the instrument was high for the 12 raters with Ebel's 
(1951) coefficient reading .92. Inter-rater reliability was also 
examined for each of the eight competencies. The 12 raters 
achieved a high degree of reliability on the eight competencies 
with Ebel's coefficient ranging from .90 to .94 (see Table 4). 

Table 4 

Eight Communication Competencies: Ebel's Coefficient 

COMPETENCY EBEL'S COEFFICIENT 

Competency One .90 

Competency Two .93 

Competency Three .91 

Competency Four .91 

Competency Five .92 

Competency Six .95 

Competency Seven .93 

Competency Eight '93 

In addition to this initial reliability test of the 
Competent Speaker form, the instrument and its criteria currently 
meet content or face validity. That validity can be argued based 
on the extensive literature review conducted during the process 
of development of the instrument by the CAT subcommittee of 11 
communication professionals. 
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Discussion 

In addition to the reliability test using 12 speech 
communicatioi! professionals, other reliability testing is being 
condactec. utilizing Graduate Teaching Assistants as raters. Also 
testing is in progress utilizing 70 faulty (non-communication 
professional) raters at a Mid-Western University. 

Regarding the validity of the instrument, there are two 
studies underway wxth regard to convergent validity: (1) a 
correlation of score on the public speaking portion of the 
Personal Report of Communication Apprehension (McCroskey, 1970) , 
with scores derived using the Competent Speaker form in an 
introductory speech class and (2) a correlation of score on the 
seven public speaking items of the Communication Competency 
Assessment Instrument (Rubin, 1982) , derived during an entrance 
interview with students in an introductory speech class, with 
scores on The Competent Speaker form. 

Future content validity testing will involve a Q-Sort being 
conducted by speech communication professionals. In that test, 
naive raters will sort all the descriptors from the criteria for 
the eight competencies, that would appear to describe the 
competency under investigation. 
Cultural Diversity 

In addition to the Competent speaker Evaluation Form and 
criteria undergoing reliability and validity scrutiny, the 
instrument also will be evaluated by a culturally diverse panel 
}f communication professionals, expert regarding intercultural 
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issues. These experts will scrutinize the instrument with regard 
to various issues related to cultural diversity and public 
speaking. Additionally, another study already underway will 
correlate the 12 speech communication professional rating of 12 
speeches using the instrument, with rating by a sample of 40 
minority students. 
Normative Data 

Although normative date has been provided elsewhere (see 
Morreale, 1991) , The Competent Speaker evaluation form and 
criteria/ standards are criterion references. The competencies 
and criteria/ standards were developed based upon the literature 
investigated. 

CONCLUSION 

The Competent Speaker form has generated much interest and 
communication professionals and instructor have expressed 
enthusiasm for its national availability. This interest may be a 
result of national trends toward accountability and attendant 
assessment in a variety of disciplines. In any case, the 
continued development and testing of the instrument appears 
warranted. 

Preliminary test of tne instrument have yielded highly 
favorable finding in regard to inter-rater reliability. Content 
validity is based on the instrument's grounding in the 
communication competency and speech evaluation literatures. 
Future directions for the instrument include additional tests 
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related to reliability, validity, and cultural diversity, to be 
conducted at a variety of academic institutions. Following such 
test, the CAT subcommittee will submit the instrument, criteria, 
training manual, and training videotape for consideration and 
distribution by the SCA Education Publication Series. Future 
research also will include the development of the a database of 
computerized critique comments to accompany the instrument. 

Generally, it is the expectation of the subcommittee that 
future tests of The Competent Speaker form will yield favorable 
findings, similar to the inter-rater reliability tests alrt>ady 
conducted. Based on such favorable tests, it is hoped that the 
instrument will be available for examination and distribution by 
SCA by Spring, 1992. 
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